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ABSTRACT 

In an effort to encourage the participation of 
educationally disadvantaged youth in the Silicon Valley's high 
technology employment boom, the Peninsula Academies program was 
established utilizing a triad-partnership arrangement among the 
Sequoia Union High School District, high technology employers in the 
area, and the Stanford Mid-Peninsula Urban Coalition (SMPUC). This 
paper examines the impact of the evaluation process on the success of 
the program, emphasizing major factors that contributed to its 
success: (1) the evaluation was adequately and independently funded; 
(2) it was designed to be useful; (3) the evaluators monitored the 
program closely, including full participation at all levels; (4) 
evaluation results were thoroughly communicated to all participants; 
and (5) the evaluation was technically sound* An evaluation, it is 
concluded, needs to be participatory and committed, working with 
program management over the entire life of a program from conception 
to replication in order to have an impact on the program's success. 
(EGS) 
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EVALUATION'S CONTRIBUTION TO THE SUCCESS OF 
A SILICON VALLEY SCHOOL/ INDUSTRY PARTNERSHIP: 
THE PENINSUU ACADEMIES EVALUATION 

The Academies Program 

The Peninsula Academies program began in 1980 as a joint effort at 
three major Silicon Valley groups that were interested in improving cppor" 
tunities for educationally disadvantaged youth, especially minority youth, 
to participate in the Valley's high technology employment boom. The groups 
involved were: (1) the Sequoia Union High School District; (2) major 
employers in the area, including Hewlett-Packard, Varian Aiisociates, 
Lockheed, and other high technology firms with facilities in the northern 
part of Silicon Valley; end (3) the Stanford Mid-Peninsula Urban Coalition 
(SMPUC). 

With SMPUC acting as the catalyst and broker, the Peninsula Academies 
program has succeeded In bringing together public and private resources to 
establish two fully operational schools-wlthln-schools, one oriented to 
electronics technology and ther other oriented toward computer applications. 
The structure of the Academies is shown in Figure 1. Thirty-five program 
graduates left the program this summer with opportunities waiting for them 
in the participating companies. The experiences of these graduates, and 
those of the 180 seniors, juniors, and sophomores now enrolled In the pro- 
gram, have Included laboratory classes taught by Industry volunteers, 
hands-on experience with modem industry-supplied equipment, special 
academic Ixxstruction that was coordinated with lab requirements, summer 
jobs, a mentoring program, and numerous other components designed to reach 
youths who were specif ical*y selected because of their liigh probability of 
dropping out of school. 

To give a better sense for the substance of the program, Attachment 1 
Illustrates th« goals and objectives of the Academies program, stated for 
all major participant groups. 
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Funded by a four-year t $150,000 gra&t froB the ViUlan aad Flora 
Hevlett Foundation in Menlo Park, staff of the Afflerican Institutes for 
Research (AIR) carried out an evaluation for the Peninsula Academies that 
was designed to promote its excellence. Ihis evaluation began in 1981-82 
with a heavy focus on problem identification and resolution; by design » the 
evaluators spent most of their early effort in (1) helping to clarify pro- 
gram goals, (2) documenting discrepancies between stated goals and actual 
implementation, and (3) troubleshooting. Only modest effort was devoted to 
collecting program outcome indicators. In 1982*83, as trouble spots became 
less numerous, more evaluation resources were expended on documenting pro- 
gram implementation and preparing for impact evaluation. By 1983-84, the 
evaluators were able to concentrate on program outcomes and the preparation 
of a Replication Guide for use by potential adopters. 

The Peninsula Academies program was recently recognized as exemplary by 
the State of California. AB 3104, sponsored by Assemblyman Bob Maylor and 
signed into law by Governor Deukmejlan, makes $1*25 million available to 
fund up to ten replications throughout the state during the forthcoming 
school year. In addition, with funding from the Edna McConnell Clark Foun- 
dation of New York, nationwide replications are currently getting underway 
in Chicago, Cleveland, Denver, Pittsburgh, and Portland. 

Commenting on che impact of the evaluation, the Peninsula Academies 
director wrote: 

I believe that it could be reasonably argued that the single 
most cost-effective component of the entire program has been 
Its evaluation. Approximately $840,000 has been spent in cash 
on the program through the end of the 1983-84 school year, with 
approximately another $1,045,000 of in-kind contributions, for 
a total of $1,885,000 (the Replication Guide provides details 
on these expenditures). The $125,000 spent so far on evalua- 
tion constitutes less than 7X of this total. Yet this 7% has 
done more to establish an effective management structure, build 
credibility, and stimulate interest in replicating the program 
than any other expenditure. It has also done more to leverage 
support from other sources,, including the state, than any other 
single feature of the program. And if the program continues to 
succeed and serve as a model for other programs, as now appears 
likely, it will have been due in substantial part to the evalu- 
ation. 



Factors Contributiii& to Evaluation Impact 

S 

Looking back over our experiences, we believe that the following five 
factors were most crucial In permitting the evaluation to make a tangible 
contribution. 

1. The evaluation was adequately and Independently funded > Evalua- 
tions are often underfunded — a truism most evaluators have learned to live 
with. While the sum of il50t000 over a four-year period is not princely, it 
has been adequate to allow us considerable freedom in addressing a wide 
range of program information needs. 

Moreover, our sponsor (the Hewlett Foundation) adopted a hands-off 
attitude from the beginning, encouraging us to develop the evaluation agenda 
and follow it as we saw fit. There was no pressure for early results, no 
wrangling over forms clearance or political protocol, no apprehension about 
the annual appropriations cycle. In short, a critical prerequisite for 
rational and orderly evaluation was in place. 

2. The evaluation was designed to be useful . Evaluations turn out 
too often to be "devali^ations,** Early on In their development, programs 
typically have unclear goals, unrealistic expectations, and confused manage- 
ment; evaluators can easily compound these difficulties by looking for 
evidence of program Impact prematurely. Although retrospective Information 
on why programs failed is not without value, ir does little service to the 
clients whom the program was designed to assist. From the beginning, the 
Peninsula Academies evaluation was designed to avoid this pitfall. Our 
early efforts were devoted almost entirely to providing assistance in 
resolving both technical and management problems as soon a^ we detected 
them > Only after we were satisfied that a program capable of producing 
Impact was in place did we begin to collect and report outcome data. 
Attachment 2 contains an overview of the evaluation activities, arranged 
according to four distinct stages of program maturation. 
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3. The evaluators "kept close to the client." We viewed our client as 
the PenloBUla Academies program, and our staff did considerable "wandering 
around" on the client's premises* From the beginning of the program, we 
attenced all meetings of the Academies Executive and Steering Committees, 
conducted systematic classro<^ observations In the fall and spring of each 
academic year, and Interviewed all major participants at the end of the 
school year In preparation for annual "Interim" evaluation reports. In 
addition, we periodically Interviewed selected students and parents (In 
their native language when necessary) and sought exit Interviews with every 
program dropout. Our goal was to achieve an In-depth understanding of the 
Academies "^rogram— -Its context, dynamics, successes, and Improvement needs. 
Attachment 3 contains a list of topics about «^lch AIR prepared Issue 
papers, In response to specific problems being encountered by Academies 
management . 

When the evaluation's focus later began to shift from operational 
assistance to documentation and valuation of outcomes, some questions were 
asked about our objectivity: how can someone so closely allied with the 
program Jse entrusted with evaluating It? Our general approach to this 
question was to let the data speak for themselves. There could be little 
debate about the quantitative outcomes that supported the emerging picture 
of program success: better attendance rates, lower school dropout rates, 
better grades, more summer employment opportunities, and better test per- 
formance by Academies students in comparison with similar nonpartlclpants. 
When coupled with Impressive process data showing the extent of industry 
participation and the enthusiasm of students and teachers, the possibility 
of "collusion" between the evaluator and program management became 
Irrelevant. 

4. The evaluation results were thoroughly communicated . We made a 
conscious effort to communicate our findings on a timely basis in forms that 
were appropriate to the intended Information users.. When Steering Committee 
members faced thorny problems, our "issue reports" attempted to clarify them, 
provide possible options, and discuss likely outcomes of each option — before 
a decision had to be made. Our Interim reports on outcomes and unresolved 
problems were written In plain English and widely distributed to all actors 
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In the prograiB* Brie£ two-page suamaries accompanied each report. When 
oajor or minor public forums were held to discuss the Academies, we were 
there to speak. When replication began to be discussed well before the 
first Academies classes had graduated* we accelerated preparation of the 
Replication Guide by a full year. Attachment 3 contains a copy of the Table 
of Contents for this Replication Guide* 

5» The evaluation was technically sound * Having been exposed over the 
years to almost all of the technical shortcomings that allow evaluations to 
be contested and set aside, we tried to design the evaluation to preempt 
methodological criticism. For example, we Invested major effort in identi- 
fying and then tracking a comparison group of students for each incoming 
Academies class. We also devoted considerable attention to the collection 
and processing of archival records and the calculation of statistics that 
converted these data Into meaningful information. Although there were no 
plans to do so, we acted as if the evaluation was going to be submitted to 
external validation by a group such as the Department of Education's Joint 
Dissemination Review Panel (JDRF). 

Conclusion 

In their book In Search of Excellence , Tom Peters and Bob Waterman 
discuss the importance of (l) single-minded commitment to detail and (2) 
an enduring central value that both drives this commitment and serves as a 
standard against which progress can be judged. We believe that these ele- 
ments also can be found in evaluations that have impact. 

It is easy to view evaluation as a detached process — cold and objective. 
But this detachment too often produces an evaluation that is sterile and 
useless. Evaluation in the service of excellence is not detached. It is 
participatory and committed, working with program management over the entire 
life of a program from conception to replication. The future of evaluation 
as a professional discipline would seem to hinge on evaluators* success in 
learning this lesson. 



ACtachiaent 1 

PENINSULA ACADEKXES 
PrAgtam Goals 

To serve an educationally disadvantaged and largely minority 
population of studentslwho toay not be succeeding in tradi- 
tional school pTOgraosJ^ toay drop outt and who lack employable 
skills* >^ 

2. To meet the vocational graining needs of such students for 
skilled and semiskilled positions with local companies » and 
to satisfy partially th^ needs of employers for employees 
in the selected fields o^ training. 



3. To confront the problem ^f youth unemployment among the target 
population, and to establish a model of possible use in other 
schools y districts, and localities. 

Program Objectives 

Students 

1. To exhibit improved attitudes toward school and learning. 

2. To exhibit improved academic performance. 

3. To exhibit improved self-*esteem» 

4. To complete the three-year Academies program and develop skills 
adequate for skilled and semi-skilled positions in electronics 
or cor.puter operations in local coti|panies. 

5. To gain knowledge of the world of i^ork and skills in choosing 

a suitable career path. j 

i 

6* After graduation, to find satisfactory employment and succeed 
on the job, or enroll In a program jof postsecondary training. 

1 

Parents j 

1. To be knowledgeable of the Academies program and its objectives 
and requirements. 

2. To support their children in the decision to enter the program 
and In their efforts, in completing the program. 

3. To assist the teachers in dealing with behavior /motivation 
problems that nay occur, 

4. To provide support to various other /program activities. 
School PereonneX (Teech.r.. Covmselors^ An^ Administrators) 

1, To contribute to the planning and definition of the program. 

2. To provide the neeeeiery fecilitiet for the Acedeniee* 

i , . ,-7- 4W' 



3. To provide an effective currlculua for che Academies, In 
conjunction with Industry and the Coalition. (Teachers) 

4. To provide academic Instruction for the program related to the 
vocational Instruction, that will lBq>rove basic skills and that 
will meet the district's proficiency objectives In English 
(reading and writing), mathematics, , and science. (Teachers) 

5. To provide students with career guidance. (Counselors) 

6. To provide the necessary high-level support within the schools 
and district and contribute to the program's long-term 
stability. (Administrators) 

Private Sector Representatives 

1. To contribute to the planning and definition of the program, 
including effective technical curricula, to ensure that students 
receive training appropriate for beginning positions. 

2. To contribute financial resources to the program. 

3. To provide t^e necessary industrial Instructors and mentors 
for the program. 

4. To help provide the necessary equipment and materials for the 
program, 

3. To provide on-site training opportunities for students. 

6. To provide employment opportunities for qualified prbgram 
graduates. 

7. To exhibit knowledge of and support for the program to others, 
expand the private sector participation in the program, and 
promote the long-term stability of the program. 

Coalition Staff 

1. To provide the necessary organization to plan and develop the ' 
program and the management and support to maintain It [for the 
field test period]. 

2. To obtain from industry those contributions necessary for the 
development and ongoing support of the program. 

3. To obtain from the schools those contributions necessary for the 
development and ongoing support of the program. 

4. To bring together industry and the schools into a working 
partnership and foster the institutionalization of this 
partnership and che program. 

5. To ensure the presence in che program of basic defining elements: 
integration of acadeadc and vocat^ional instruction, individualiza- 
tion of instruction, aocivacional componencs, and career counseling* 



To develop and u«e an effeccive aoniCorlng and feedback 
•ytCCB for th« progroi (in conjuaccion with the schools). 
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ROUES OF THE ASSISTANCE/EVALUATION CONTRACTOR 
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MSTtTUnONAUZATION STAGE 
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Actacbnent 3 

Issue» Addres»»d by Spedtl Evaluating Rgpgrts 

/ 

• Defining student selection criteria and proce^kures 

• Development of currlculun 

• Defining a workable program management structure 

• Defining acceptable and measurable program goals and objectives 

• Defining the roles of the various coimittees, and hov they fit 
together 

• Addressing the Issue of esiployment skills and their measurement 

• Developing coordination between the Academies and the general 
school program 

• Broadening the industry support base 

• Refining program decision-making mechanisms 

• Evolving prjccdurescfrr dealing with problem students, including 
expulsion cz;Lterir, and procedures 

• Identifying appropriate entry- level jobs, especially in the 
computer area 

• Integrating career counseling 

• Defining staff orientation and development needs 

• Providing role clarification for the school. Coalition, and 
cotnmittee representatives 

• Refining the program's organizational model and planning for 
Institutionalization 
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